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The  Geographic  Resources  Analysis  Support  System 
(GRASS)  is  a  geographic  information  and  image  processing 
system  originally  designed  to  serve  land  managers  and 
environment  planners  at  Army  installations.  GRASS  is 
public  domain  software  distributed  by  several  public  and 
private  organizations.  The  GRASS  Inter-Agency  Steering 
Committee  represents  the  organizations  that  want  to  ensure 
that  GRASS  software  remain  reliable,  consistent,  and  effi¬ 
cient,  and  that  new  versions  of  the  code,  new  hardware 
platforms,  and  new  digitizer,  monitor,  and  printer  drivers  be 
consistently  tested  before  distribution.  This  report  provides 
guidance  for  testing  new  hardware  platforms  and  drivers  for 
GRASS. 

To  be  fully  supported,  hardware  and  software  configurations 
must  pass  both  alpha  and  beta  testing.  Some  GRASS 
software  distribution  site  must  accept  responsibility  for 
support  of  new  hardware  configurations  or  drivers.  Alpha 
testing  is  usually  done  internally,  after  an  initial  port  of 
GRASS,  by  an  organization  that  has  created  a  new  port  or 
driver.  During  alpha  testing,  a  serious  effort  is  made  to 
identify  and  correct  problems  resulting  from  new  code  by  a 
coordinated  effort  between  the  programmers  and  testers. 
The  test  is  initiated  by  the  test  coordinator,  conducted  by  the 
beta  test  sites,  and  verified  by  the  GRASS  hardware  coordi¬ 
nator.  The  end  product  is  fully  documented  computer  code. 
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FOREWORD 
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Testing  Guidelines 
for  GRASS  Ports  and  Drivers 


by  William  D.  Goran 
GRASS  Information  Center 

U.S.  Army  Construction  Engineering  Research  Laboratory 
P.0.  Box  4005,  Champaign,  IL  61824-4005 
Phone  217/373-7220  or  800/USA-CERL  extension  220 


1.  Introduction 

GRASS,  the  Geographic  Resources  Analysis  Support  System,  is  public  domain 
software  distributed  by  several  public  and  private  organizations.  The  organiza¬ 
tions  that  develop  and  support  GRASS  (represented  by  the  GRASS  Inter- 
Agency  Steering  Committee)  want  to  insure  that  the  software  is  reliable,  con 
sistent,  and  efficient,  and  that  new  versions  of  the  code;  ports  to  new  hardware 
platforms;  and  new  digitizer,  monitor  and  printer  drivers  are  reasonably  and 
consistently  tested  before  general  distribution.  This  document  provides  gui¬ 
dance  for  testing  new  hardware  platforms  and  drivers. 

The  GRASS  Hardware  Configurations  Guide 

The  GRASS  Hardware  Configurations  Guide1 2  identifies  all  hardware  configurations  that  support 
GRASS  software,  and  all  peripherals  for  which  digitizer,  monitor,  or  printer  drivers  have  been 
written.  There  are  two  configuration  sections  to  this  guide  -  (1)  configurations  that  are  fully 
tested  and  supported  for  GRASS,  and  (2)  configurations  that  are  in  beta  testing.  For  hardware 
configurations  (including  new  drivers)  to  move  from  the  beta  testing  section  to  the  tested  and 
supported  section,  the  beta  test  program  must  be  completed,  and  some  site(s)  that  distributes 
GRASS  software  must  accept  responsibility  for  support  of  this  new  configuration  and/or  driver. 

The  GRASS  Hardware  Configurations  Guide  is  maintained  and  updated  by  the  U.S.  Army  Con¬ 
struction  Engineering  Research  Laboratory  (USACERL)  GRASS  Hardware  Coordinator  (Doug 
Brooks) .  Beta  test  sponsors  should  notify  the  GRASS  Hardware  Coordinator  at  least  twice  dur¬ 
ing  this  testing  process: 

•  When  a  new  configuration  or  driver  is  ready  for  testing 
(Beta  Test  Initiation  Notice) . 

•  When  a  beta  test  has  been  successfully  completed 
(Test  Completion  Report). 

Configuration  information  and  benchmarks  should  be  provided  to  to  the  GRASS  Hardware 
Coordinator  both  before  and  after  (if  there  are  any  changes)  the  beta  testing  process.  Bench¬ 
marks  should  follow  the  instructions  in  the  document  entitled  Guidelines  for  Running  GRASS 
Benchmarks1  authored  by  Mark:  Johnson  of  USACERL.  These  documents  are  available  upon 
request  from  the  GRASS  Information  Center.  The  Center  can  also  provide  contacts  for  all  sites 

1  Douglas  A.  Brooks,  Michael  E.  Higgins,  and  Mark  O.  Johnson,  GRASS  Hardware  Configurations  Guide, 

ADP  Report  N-89/21  (U.S.  Army  Construction  Engineering  Research  Laboratory  [USACERL),  March 
1969). 

2  Mark  Johnson,  Guidelines  for  Running  GRASS  Benchmarks,  Technical  Manuscript  N-89/23  (USACERL, 

February  1969). 
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that  distribute  or  support  GRASS  software,  or  that  are  currently  involved  in  beta  tests. 

2.  Alpha  Testing 

After  an  initial  port  of  GRASS  has  first  been  completed,  or  a  new  driver  written,  alpha  testing 
should  be  conducted.  A  significant  effort  should  be  made,  during  alpha  testing,  to  identify  and 
correct  problems  and  limitations  resulting  from  the  new  code.  Alpha  testing  should  also 
include  running  the  initial  GRASS  benchmarks. 

There  are  several  characteristics  that  distinguish  alpha  from  beta  testing. 

•  Alpha  testing  precedes  beta  testing. 

•  During  alpha  testing,  programmers  and  testers  work  in  close  coordination,  and  programmers 
can  change  code  to  correct  problems  on  a  frequent  and  informal  basis,  at  the  request  of  the  tes¬ 
ters. 

•  No  written  documentation  is  required  for  the  alpha  testing  process. 

•  Usually  alpha  testing  is  internal,  within  the  organization  that  created  the  new  port  or  driver. 

In  contrast,  the  beta  test  follows  the  alpha  test,  involves  sites  external  to  the  originating  organi¬ 
zation,  requires  formal  notification  and  documentation,  and  changes  should  only  be  made  to  the 
code  after  the  completion  of  the  testing  process. 

3.  Beta  Testing 

During  alpha  testing,  plans  should  be  made  for  the  beta  testing  process.  The  following  are 
guidelines  to  provide  developers,  supporters,  testers  and  users  with  a  set  of  "common”  pro¬ 
cedures.  There  are  three  types  of  participants  in  this  beta  testing  process,  ( 1)  the  test  coordina¬ 
tor,  (2)  the  beta  testing  sites,  and  (3)  the  GRASS  Hardware  Coordinator.  The  beta  testing  is 
initiated  by  the  test  coordinator,  conducted  by  the  beta  test  sites,  and  verified,  both  before  and 
after  the  testing,  by  the  GRASS  Hardware  Coordinator. 

Generally,  there  should  be  at  least  three  beta  test  sites,  although  in  some  circumstances  this 
number  will  be  greater  or  smaller.  Any  decision  to  use  less  than  three  sites  should  be  approved 
by  the  GRASS  Hardware  Coordinator.  Sites  selected  for  beta  testing  should  already  be  familiar 
with  GRASS  software. 


The  Beta  Test  Coordinator 

Beta  tests  will  be  coordinated  by  a  angle  site,  usually  the  developer  or  sponsor  of  the  new 
driver  or  port.  This  site  will  be  called  the  Beta  Test  Coordinator.  The  Beta  Test  Coordinator 
will  prepare  three  documents  before  the  beta  testing  begins.  These  documents  are  (1) 
Configuration  Specifications,  (2)  Installation  Instructions,  and  (3)  Test  Report  Forms. 

<  1)  Configuration  Specifications: 

Exact  hardware  configuration  information  should  be  provided  by  the  test  coordinator  to  all 
potential  beta  test  candidates,  and  to  the  GRASS  Hardware  Coordinator.  This  information 
should  include  details  on  all  elements  of  a  configuration,  including  memory,  disk,  storage 
devices,  special  boards  and  any  other  requirements.  For  new  ports,  information  should 
also  be  provided  on  which  peripherals  were  tested,  and  information  on  how  there  peri¬ 
pherals  were  connected.  For  driver  ports,  information  should  be  provided  as  to  which 
computers  and  operating  systems  (and  releases  of  operating  systems)  and  compilers  were 
used  dining  development  of  a  driver,  and  any  special  switch  settings  and  cable 
configurations  relevant  to  the  operation  of  the  printer,  digitizer  or  other  device.  When  for¬ 
warded  to  the  GRASS  Hardware  Coordinator,  configuration  information  should  include 
configuration  options  (e.g.  comparable  workstation  in  a  series,  such  as  the  Sun  386i  150 
and  the  Sun  386i  250)  and  list  prices  for  each  item  in  the  configuration. 
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(2)  Installation  Instructions: 

Full  installation  instructions  should  be  developed  by  the  test  coordinator  before  tests  can 
be  conducted.  Installing  the  software  on  a  new  machine  is  one  of  the  key  elements  in  the 
test.  Also,  test  coordinators  should  identify  any  changes  that  might  be  required  to  stan¬ 
dard  GRASS  user  or  programmer  documentation,  because  of  this  new  hardware.  ( Such 
changes  should  be  minimal,  to  insure  portability  of  GRASS  software  between  different 
hardware  platforms).  Installation  documentation  is  already  included  for  some  hardware 
platforms  in  the  GRASS  User’s  Reference  Manual? .  An  addendum  to  these  existing  instal¬ 
lation  instructions  may  be  sufficient  However,  in  some  cases,  installation  instructions  will 
need  to  discuss  such  matters  as  the  insertion  of  new  boards  into  a  hardware  bus.  The 
installation  instructions  should  also  identify  any  software  requirements  for  GRASS  on  the 
hardware  platform,  such  as  GKS,  Sun  view,  X  or  other  graphic  libraries. 

(3)  Test  Report  Fbrms: 

This  form  is  simply  a  listing  of  all  programs  to  be  tested,  with  places  for  the  testing  sites  to 
check  each  program  and  note  any  problems.  For  new  ports,  all  programs  in  the  current 
release  version  of  GRASS  should  be  tested  For  new  drivers,  all  GRASS  programs  and 
functions  within  programs  that  relate  to  the  device  (e.g.  digitizer  or  printer  or  plotter) 
should  be  tested.  This  form  is  to  be  returned  at  the  completion  of  the  testing  process. 

Some  public  notice  should  be  made,  if  possible,  to  inform  potential  beta  test  candidates  that  a 
product  will  be  available  for  beta  testing.  This  notice  should  identify  the  organization  conduct¬ 
ing  this  test,  the  intended  timeframe  for  and  duration  of  the  test,  and  the  anticipated  require¬ 
ments  (e.g.  hardware  configurations  and/or  software  or  technical  expertise)  of  potential  sites. 
Forums  for  such  announcements  include  [1]  GRASSClippings  newsletter,4  [2]  direct  mailing  to 
GRASS  user  sites,  [3]  technical  journals  and  meetings  and  [4]  GRASSNET  and  other  electronic 
notes/bulletin  boards. 

Frequently,  hardware  vendors  will  help  facilitate  beta  testing,  and  some  vendors  will  provide 
equipment  to  selected  beta  testing  sites.  It  is  recommended  that  Beta  Test  Coordinators 
involve  the  vendors,  to  whatever  extend  appropriate,  to  confirm  information  on  configurations, 
pricing  and  hardware  availability  and  to  develop  plans  for  software  support.  Arrangements  with 
vendors,  however,  are  the  primary  responsibility  of  the  Beta  Test  Coordinators,  not  the  testing 
sites  or  the  GRASS  Hardware  Coordinator. 

Before  the  code  is  released  to  the  testing  sites,  the  Beta  Test  Coordinator  should  send  a  Beta 
Test  Initiation  Notice  to  the  GRASS  Hardware  Coordinator  identifying  the  intent  to  begin  the 
test,  the  equipment  to  be  tested,  list  prices  for  each  item  in  the  configuration,  initial  bench¬ 
marks  (if  appropriate),  any  vendor  arrangements,  and  the  names  and  points  of  contact  for  beta 
testing  sites.  The  GRASS  Hardware  Coordinator  will  then  list  this  configuration,  and  the  test 
coordinator,  in  the  next  version  of  the  GRASS  Hardware  Configurations  Guide. 

When  the  beta  testing  is  completed,  all  software  bugs  and  documentation  errors,  identified  dur¬ 
ing  the  beta  testing,  should  be  addressed  and  corrected,  before  the  code  and  documentation  are 
released.  However,  no  new  functions  should  be  added  between  beta  testing  and  final  release. 
If  critical  functions  were  omitted,  they  should  be  added  and  retested.  If  non-critical  functions 
are  desired,  they  should  be  included  in  future  software  releases  or  driver  enhancements. 

Then,  the  Beta  Test  Coordinator  should  file  a  Test  Completion  Report  with  the  GRASS  Hardware 
Coordinator.  This  report  should  include  ( 1)  copies  of  the  Test  Report  Fbrms  from  each  beta  test 

®  James  Westervelt,  Michael  Shapiro,  William  D.  Goran,  et  aL,  GRASS  User’s  Reference  Manual,  ADP  Re¬ 
port  N-87/22  (USACERL,  September  1988). 

4  GRASSClippings,  ISSN  0899-7863,  published  quarterly  by  the  GRASS  Inter-Agency  Steering  Committee. 
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site,  (2)  confirmation  that  each  problem  identified  by  the  testing  sites  has  been  identified  and 
corrected,  (3)  any  new  relevant  configuration  or  benchmark  information,  and  (4)  plans  for  dis¬ 
tribution  and  support  of  this  new  hardware.  Distribution  of  code  for  this  new  hardware  should 
not  begin  until  the  GRASS  Hardware  Coordinator  confirms  to  the  Beta  Test  Coordinator  receipt 
and  acceptance  of  this  Beta  Test  Completion  Report. 

Beta  Testing  Sites 

Sites  that  wish  to  participate  in  beta  tests  should  notify  the  Beta  Test  Coordinator.  In  this 
request  notice,  these  sites  will  need  to  verify  that  they  have  the  correct  equipment,  necessary 
skills  ( knowledge  of  UNIX,  GRASS,  and  the  hardware)  and  resources  to  perform  this  test. 

Beta  tests  must  be  performed  on  the  same  or  equivalent  hardware  as  was  used  for  developing 
the  software.  Testing  software  on  different  hardware  constitutes  "porting”  the  code,  and  bugs 
discovered  during  this  process  could  result  from  the  software  port  or  the  original  code.  As  the 
beta  test  is  designed  to  discover  (and  correct)  bugs  in  the  original  code,  testing  should  be  kept 
as  simple  as  possible.  Once  the  original  driver  or  port  is  fully  tested,  tests  of  hardware  varia¬ 
tions  might  also  be  conducted. 

The  beta  testing  procedures  are  as  follows: 

1.  Upon  receiving  the  new  software,  review  the  installation  instructions,  and  begin  the  software 
installation  process.  Note  any  problems  with  the  procedures  or  the  documentation. 

2.  Perform  a  systematic  testing  of  each  program  listed  in  the  Test  Report,  and  note  any  prob¬ 
lems  encountered.  There  notes  should  include  details  about  any  system  error  messages. 

3.  After  testing  the  programs,  run  the  GRASS  benchmarks  to  confirm  the  Test  Coordinators 
results  and  identify  any  performance  problems. 

4.  Make  every  effort  to  complete  the  testing  process,  and  return  the  test  results,  within  the 
specified  timeframe.  Often,  the  Test  Coordinator  is  under  time  pressure  to  have  the  test  com¬ 
pleted  and  to  begin  distribution  of  the  code. 

GRASS  Hardware  Coordinator 

Before  the  testing  begins,  the  GRASS  Hardware  Coordinator  (GHC)  will  need  to  verify  that  the 
information  from  the  Beta  Test  Coordinator  is  correct  Then,  the  GHC  will  update  the  beta 
testing  section  of  the  GRASS  Hardware  Configurations  Guide,  and  notify  the  editor  of 
GRASSCUppings  about  the  planned  test.  Next  the  Hardware  Coordinator  will  contact  each  test¬ 
ing  site,  and  confirm,  with  the  point  of  contact  that  the  testing  procedures  are  understood. 

When  the  testing  is  completed,  the  GHC  should  receive  and  review  the  Completion  Report 
from  the  Beta  Test  Coordinator,  and  confirm  from  the  report  contents  that  all  known  problems 
have  been  corrected.  Then,  the  benchmarks  and  configurations  should  be  updated,  and  the 
new  configuration  should  be  moved  to  the  "tested  and  supported"  section  of  the  hardware 
configuration  guide.  The  Beta  Test  Coordinator  needs  to  arrange  for  ongoing  support  of  the 
new  port  or  driver,  through  an  existing  or  new  support  organization.5  Finally,  a  notice  of  the 
completed  port  should  be  made  in  GRASSCUppings,  and  the  GRASS  Information  Center  records 
should  be  updated. 


8  If  there  is  a  new  support  organization,  some  memorandum  of  agreement  might  be  required  between 
USA C ERL  (on  behalf  of  the  GRASS  Inter-Agency  Steering  Committee)  and  the  potential  distribution  site. 
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